Skip to content

numa: discover NUMA topology from ACPI SRAT/SLIT - #1418

Closed
gburd wants to merge 1 commit into
cloudius-systems:masterfrom
gburd:pr/numa-discovery
Closed

numa: discover NUMA topology from ACPI SRAT/SLIT#1418
gburd wants to merge 1 commit into
cloudius-systems:masterfrom
gburd:pr/numa-discovery

Conversation

@gburd

@gburd gburd commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

What

OSv has no notion of NUMA: the scheduler and allocator treat memory as flat,
which leaves performance on the table on multi-socket bare-metal (the large
EC2/GCP metal instances that are a target for Postgres-over-ZFS workloads).

This is the first, discovery-only step of NUMA support. It adds a numa::
module that parses the ACPI SRAT (System Resource Affinity Table) and SLIT
(System Locality Distance Information Table) at boot, right after
acpi::init(), and exposes the topology.

API

  • numa::nr_nodes() / numa::available()
  • numa::node_of_cpu(cpu_id) -- resolved by correlating SRAT APIC ids with the
    APIC ids the MADT parse already recorded on each sched::cpu
  • numa::distance(from, to) -- from SLIT (10 == local, per ACPI convention),
    defaulting to 10 local / 20 remote when no SLIT is present
  • numa::memory_ranges() -- physical ranges tagged with their node

It handles the SRAT CPU-affinity, x2APIC CPU-affinity and memory-affinity
subtables, and only trusts SLIT if its locality count matches the node count
SRAT reported. On a machine with no SRAT (the common single-node VM) it reports
one flat node and available() == false.

Scope

This intentionally does not yet change allocation or scheduling; it only
makes the topology available so later work (a node-aware allocator, scheduler
affinity, mbind/get_mempolicy) can build on it. Nothing changes behavior on
existing single-node setups.

Testing

tests/tst-numa.cc validates the invariants (>= 1 node, every CPU maps in
range, distance diagonal == 10 and off-diagonal >= 10, memory ranges name valid
nodes). Verified on OSv under KVM both without NUMA (reports 1 flat node) and
with a QEMU 2-node -numa config (reports 2 nodes, 3 memory ranges,
available).

OSv had no notion of NUMA: the scheduler and allocator treat memory as flat,
which leaves performance on the table on multi-socket bare-metal (the large
EC2/GCP metal instances that are a target for Postgres-over-ZFS).

This is the first, discovery-only step. It adds a numa:: module that parses the
ACPI SRAT (System Resource Affinity Table) and SLIT (System Locality Distance
Information Table) at boot, right after acpi::init(), and exposes the topology:

- numa::nr_nodes() / numa::available()
- numa::node_of_cpu(cpu_id) -- resolved by correlating SRAT APIC ids with the
  APIC ids the MADT parse recorded on each sched::cpu
- numa::distance(from, to) -- from SLIT (10 == local per ACPI convention),
  defaulting to 10 local / 20 remote when no SLIT is present
- numa::memory_ranges() -- physical ranges tagged with their node

It handles the SRAT CPU-affinity, x2APIC CPU-affinity and memory-affinity
subtables, and only trusts SLIT if its locality count matches the node count
SRAT reported. On a machine with no SRAT (the common single-node VM) it reports
one flat node and available() == false; nothing changes behavior.

This intentionally does NOT yet change allocation or scheduling; it only makes
the topology available so a node-aware allocator, scheduler affinity, and
mbind/get_mempolicy can build on it.

Add tests/tst-numa.cc validating the invariants (>= 1 node, every CPU maps in
range, distance diagonal == 10 and off-diagonal >= 10, memory ranges name valid
nodes). Verified on OSv under KVM both without NUMA (reports 1 flat node) and
with a QEMU 2-node -numa config (reports 2 nodes, 3 memory ranges, available).
@gburd

gburd commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #1437 (rebased onto current master; GitHub auto-closed this one when the branch was updated). Please review #1437 instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant